Ghrelin decreases sensitivity to negative feedback and increases prediction-error related caudate activity in humans, a randomized controlled trial.

The stomach-derived hormone ghrelin plays not only a role in feeding, starvation, and survival, but it has been suggested to also be involved in the stress response, in neuropsychiatric conditions, and in alcohol and drug use disorders. Mechanisms related to reward processing might mediate ghrelin's broader effects on complex behaviors, as indicated by animal studies and mostly correlative human studies. Here, using a within-subject double-blind placebo-controlled design with intravenous ghrelin infusion in healthy volunteers (n = 30), we tested whether ghrelin alters sensitivity to reward and punishment in a reward learning task. Parameters were derived from a computational model of participants' task behavior. The reversal learning task with monetary rewards was performed during functional brain imaging to investigate ghrelin effects on brain signals related to reward prediction errors. Compared to placebo, ghrelin decreased punishment sensitivity (t = -2.448, p = 0.021), while reward sensitivity was unaltered (t = 0.8, p = 0.43). We furthermore found increased prediction-error related activity in the dorsal striatum during ghrelin administration (region of interest analysis: t-values ≥ 4.21, p-values ≤ 0.044). Our results support a role for ghrelin in reward processing that extends beyond food-related rewards. Reduced sensitivity to negative outcomes and increased processing of prediction errors may be beneficial for food foraging when hungry but could also relate to increased risk taking and impulsivity in the broader context of addictive behaviors.

Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks.

Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.

Optogenetic stimulation of corticostriatal circuits improves behavioral flexibility in mice with prenatal alcohol exposure.

Fetal alcohol spectrum disorder (FASD) is the most common preventable form of developmental and neurobehavioral disability. Animal models have demonstrated that even low to moderate prenatal alcohol exposure (PAE) is sufficient to impair behavioral flexibility in multiple domains. Previously, utilizing a moderate limited access drinking in the dark paradigm, we have shown that PAE 1) impairs touchscreen pairwise visual reversal in male adult offspring 2) leads to small but significant decreases in orbitofrontal (OFC) firing rates 3) significantly increases dorsal striatum (dS) activity and 4) aberrantly sustains OFC-dS synchrony across early reversal. In the current study, we examined whether optogenetic stimulation of OFC-dS projection neurons would be sufficient to rescue the behavioral inflexibility induced by PAE in male C57BL/6J mice. Following discrimination learning, we targeted OFC-dS projections using a retrograde adeno-associated virus (AAV) delivered to the dS which expressed channel rhodopsin (ChR2). During the first four sessions of reversal learning, we delivered high frequency optogenetic stimulation to the OFC via optic fibers immediately following correct choice responses. Our results show that optogenetic stimulation significantly reduced the number of sessions, incorrect responses, and correction errors required to move past the early perseverative phase for both PAE and control mice. In addition, OFC-dS stimulation during early reversal learning reduced the increased sessions, correct and incorrect responding seen in PAE mice during the later learning phase of reversal but did not significantly alter later performance in control ChR2 mice. Taken together these results suggest that stimulation of OFC-dS projections can improve early reversal learning in PAE and control mice, and these improvements can persist even into later stages of the task days later. These studies provide an important foundation for future clinical approaches to improve executive control in those with FASD. This article is part of the Special Issue on "PFC circuit function in psychiatric disease and relevant models".

Dynamic regulation of corticostriatal glutamatergic synaptic expression during reversal learning in male mice.

Behavioral flexibility, one of the core executive functions of the brain, has been shown to be an essential skill for survival across species. Corticostriatal circuits play a critical role in mediating behavioral flexibility. The molecular mechanisms underlying these processes are still unclear. Here, we measured how synaptic glutamatergic α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptor (AMPAR) and N-methyl-D-aspartic acid receptor (NMDAR) expression dynamically changed during specific stages of learning and reversal. Following training to well-established stages of discrimination and reversal learning on a touchscreen visual task, lateral orbitofrontal cortex (OFC), dorsal striatum (dS) as well as medial prefrontal cortex (mPFC), basolateral amygdala (BLA) and piriform cortex (Pir) were micro dissected from male mouse brain and the expression of glutamatergic receptor subunits in the synaptic fraction were measured via immunoblotting. We found that the GluN2B subunit of NMDAR in the OFC remained stable during initial discrimination learning but significantly increased in the synaptic fraction during mid-reversal stages, the period during which the OFC has been shown to play a critical role in updating outcome expectancies. In contrast, both GluA1 and GluA2 subunits of the AMPAR significantly increased in the dS synaptic fraction as new associations were learned late in reversal. Expression of NMDAR and AMPAR subunits did not significantly differ across learning stages in any other brain region. Together, these findings further support the involvement of OFC-dS circuits in moderating well-learned associations and flexible behavior and suggest that dynamic synaptic expression of NMDAR and AMPAR in these circuits may play a role in mediating efficient learning during discrimination and the ability to update previously learned associations as environmental contingencies change.

Dopamine release in human associative striatum during reversal learning.

The dopaminergic system is firmly implicated in reversal learning but human measurements of dopamine release as a correlate of reversal learning success are lacking. Dopamine release and hemodynamic brain activity in response to unexpected changes in action-outcome probabilities are here explored using simultaneous dynamic [11C]Raclopride PET-fMRI and computational modelling of behavior. When participants encounter reversed reward probabilities during a card guessing game, dopamine release is observed in associative striatum. Individual differences in absolute reward prediction error and sensitivity to errors are associated with peak dopamine receptor occupancy. The fMRI response to perseverance errors at the onset of a reversal spatially overlap with the site of dopamine release. Trial-by-trial fMRI correlates of absolute prediction errors show a response in striatum and association cortices, closely overlapping with the location of dopamine release, and separable from a valence signal in ventral striatum. The results converge to implicate striatal dopamine release in associative striatum as a central component of reversal learning, possibly signifying the need for increased cognitive control when new stimuli-responses should be learned.

Comparable roles for serotonin in rats and humans for computations underlying flexible decision-making.

Serotonin is critical for adapting behavior flexibly to meet changing environmental demands. Cognitive flexibility is important for successful attainment of goals, as well as for social interactions, and is frequently impaired in neuropsychiatric disorders, including obsessive-compulsive disorder. However, a unifying mechanistic framework accounting for the role of serotonin in behavioral flexibility has remained elusive. Here, we demonstrate common effects of manipulating serotonin function across two species (rats and humans) on latent processes supporting choice behavior during probabilistic reversal learning, using computational modelling. The findings support a role of serotonin in behavioral flexibility and plasticity, indicated, respectively, by increases or decreases in choice repetition ('stickiness') or reinforcement learning rates following manipulations intended to increase or decrease serotonin function. More specifically, the rate at which expected value increased following reward and decreased following punishment (reward and punishment 'learning rates') was greatest after sub-chronic administration of the selective serotonin reuptake inhibitor (SSRI) citalopram (5 mg/kg for 7 days followed by 10 mg/kg twice a day for 5 days) in rats. Conversely, humans given a single dose of an SSRI (20 mg escitalopram), which can decrease post-synaptic serotonin signalling, and rats that received the neurotoxin 5,7-dihydroxytryptamine (5,7-DHT), which destroys forebrain serotonergic neurons, exhibited decreased reward learning rates. A basic perseverative tendency ('stickiness'), or choice repetition irrespective of the outcome produced, was likewise increased in rats after the 12-day SSRI regimen and decreased after single dose SSRI in humans and 5,7-DHT in rats. These common effects of serotonergic manipulations on rats and humans-identified via computational modelling-suggest an evolutionarily conserved role for serotonin in plasticity and behavioral flexibility and have clinical relevance transdiagnostically for neuropsychiatric disorders.

Dissociable Contributions of Basolateral Amygdala and Ventrolateral Orbitofrontal Cortex to Flexible Learning Under Uncertainty.

Reversal learning measures the ability to form flexible associations between choice outcomes with stimuli and actions that precede them. This type of learning is thought to rely on several cortical and subcortical areas, including the highly interconnected orbitofrontal cortex (OFC) and basolateral amygdala (BLA), and is often impaired in various neuropsychiatric and substance use disorders. However, the unique contributions of these regions to stimulus- and action-based reversal learning have not been systematically compared using a chemogenetic approach particularly before and after the first reversal that introduces new uncertainty. Here, we examined the roles of ventrolateral OFC (vlOFC) and BLA during reversal learning. Male and female rats were prepared with inhibitory designer receptors exclusively activated by designer drugs targeting projection neurons in these regions and tested on a series of deterministic and probabilistic reversals during which they learned about stimulus identity or side (left or right) associated with different reward probabilities. Using a counterbalanced within-subject design, we inhibited these regions prior to reversal sessions. We assessed initial and pre-/post-reversal changes in performance to measure learning and adjustments to reversals, respectively. We found that inhibition of the ventrolateral orbitofrontal cortex (vlOFC), but not BLA, eliminated adjustments to stimulus-based reversals. Inhibition of BLA, but not vlOFC, selectively impaired action-based probabilistic reversal learning, leaving deterministic reversal learning intact. vlOFC exhibited a sex-dependent role in early adjustment to action-based reversals, but not in overall learning. These results reveal dissociable roles for BLA and vlOFC in flexible learning and highlight a more crucial role for BLA in learning meaningful changes in the reward environment.

Meta-reinforcement learning via orbitofrontal cortex.

The meta-reinforcement learning (meta-RL) framework, which involves RL over multiple timescales, has been successful in training deep RL models that generalize to new environments. It has been hypothesized that the prefrontal cortex may mediate meta-RL in the brain, but the evidence is scarce. Here we show that the orbitofrontal cortex (OFC) mediates meta-RL. We trained mice and deep RL models on a probabilistic reversal learning task across sessions during which they improved their trial-by-trial RL policy through meta-learning. Ca2+/calmodulin-dependent protein kinase II-dependent synaptic plasticity in OFC was necessary for this meta-learning but not for the within-session trial-by-trial RL in experts. After meta-learning, OFC activity robustly encoded value signals, and OFC inactivation impaired the RL behaviors. Longitudinal tracking of OFC activity revealed that meta-learning gradually shapes population value coding to guide the ongoing behavioral policy. Our results indicate that two distinct RL algorithms with distinct neural mechanisms and timescales coexist in OFC to support adaptive decision-making.

Reversal learning in those with early psychosis features contingency-dependent changes in loss response and learning.

INTRODUCTION: People with psychotic disorders commonly feature broad decision-making impairments that impact their functional outcomes. Specific associative/reinforcement learning problems have been demonstrated in persistent psychosis. But these phenotypes may differ in early psychosis, suggesting that aspects of cognition decline over time. METHODS: The present proof-of-concept study examined goal-directed action and reversal learning in controls and those with early psychosis. RESULTS: Equivalent performance was observed between groups during outcome-specific devaluation, and reversal learning at an 80:20 contingency (reward probability for high:low targets). But when the low target reward probability was increased (80:40) those with early psychosis altered their response to loss, whereas controls did not. Computational modelling confirmed that in early psychosis there was a change in punishment learning that increased the chance of staying with the same stimulus after a loss, multiple trials into the future. In early psychosis, the magnitude of this response was greatest in those with higher IQ and lower clinical severity scores. CONCLUSIONS: We show preliminary evidence that those with early psychosis present with a phenotype that includes altered responding to loss and hyper-adaptability in response to outcome changes. This may reflect a compensatory response to overcome the milieu of corticostriatal changes associated with psychotic disorders.

Explaining reversal learning deficits in anxiety with electrophysiological evidence.

Reversal learning is a crucial aspect of behavioral flexibility that plays a significant role in environmental adaptation and development. While previous studies have established a link between anxiety and impaired reversal learning ability, the underlying mechanisms behind this association remain unclear. This study employed a probabilistic reversal learning task with electroencephalographic recording to investigate these mechanisms. Participants were divided into two groups based on their scores on Spielberger's State-Trait Anxiety Inventory: high trait-anxiety (HTA) and low trait-anxiety (LTA), consisting of 50 individuals in each group. The results showed that the HTA group had poorer reversal learning performance than the LTA group, including a lower tendency to shift to the new optimal option after rule reversals (reversal-shift). The study also examined event-related potentials elicited by reversals and found that although the N1 (related to attention allocation), feedback-related negativity (FRN: related to belief updating), and P3 (related to response inhibition) were all sensitive to the grouping factor, only the FRN elicited by reversal-shift mediated the relationship between anxiety and the number/reaction time of reversal-shift. From these findings, we suggest that abnormalities in belief updating may contribute to the impaired reversal learning performance observed in anxious individuals. In our opinion, this study sheds light on potential targets for interventions aimed at improving behavioral flexibility in anxious individuals.

Chronic corticosterone improves perseverative behavior in mice during sequential reversal learning.

BACKGROUND: Stressful life events can both trigger development of psychiatric disorders and promote positive behavioral changes in response to adversities. The relationship between stress and cognitive flexibility is complex, and conflicting effects of stress manifest in both humans and laboratory animals. OBJECTIVE: To mirror the clinical situation where stressful life events impair mental health or promote behavioral change, we examined the post-exposure effects of stress on cognitive flexibility in mice. METHODS: We tested female C57BL/6JOlaHsd mice in the touchscreen-based sequential reversal learning test. Corticosterone (CORT) was used as a model of stress and was administered in the drinking water for two weeks before reversal learning. Control animals received drinking water without CORT. Behaviors in supplementary tests were included to exclude non-specific confounding effects of CORT and improve interpretation of the results. RESULTS: CORT-treated mice were similar to controls on all touchscreen parameters before reversal. During the low accuracy phase of reversal learning, CORT reduced perseveration index, a measure of perseverative responding, but did not affect acquisition of the new reward contingency. This effect was not related to non-specific deficits in chamber activity. CORT increased anxiety-like behavior in the elevated zero maze test and repetitive digging in the marble burying test, reduced locomotor activity, but did not affect spontaneous alternation behavior. CONCLUSION: CORT improved cognitive flexibility in the reversal learning test by extinguishing prepotent responses that were no longer rewarded, an effect possibly related to a stress-mediated increase in sensitivity to negative feedback that should be confirmed in a larger study.

Two-stage reinforcement learning task predicts psychological traits.

External sources of information influence human actions. However, psychological traits (PTs), considered internal variables, also play a crucial role in decision making. PTs are stable across time and contexts and define the set of behavioral repertoires that individuals express. Here, we explored how multiple metrics of adaptive behavior under uncertainty related to several PTs. Participants solved a reversal-learning task with volatile contingencies, from which we characterized a detailed behavioral profile based on their response sequences. We then tested the relationship between this multimetric behavioral profile and scores obtained from self-report psychological questionnaires. The PT measurements were based on the Hierarchical Taxonomy Of Psychopathology (HiTOP) model. By using multiple linear regression models (MLRMs), we found that the learning curves predicted important differences in the PTs and task response times. We confirmed the significance of these relationships by using random permutations of the predictors of the MLRM. Therefore, the behavioral profile configurations predicted the PTs and served as a "fingerprint" to identify participants with a high certainty level. We discuss briefly how this characterization and approach could contribute to better nosological classifications.

How uncertain are you? Disentangling expected and unexpected uncertainty in pupil-linked brain arousal during reversal learning.

During decision making, we are continuously faced with two sources of uncertainty regarding the links between stimuli, our actions, and outcomes. On the one hand, our expectations are often probabilistic, that is, stimuli or actions yield the expected outcome only with a certain probability (expected uncertainty). On the other hand, expectations might become invalid due to sudden, unexpected changes in the environment (unexpected uncertainty). Several lines of research show that pupil-linked brain arousal is a sensitive indirect measure of brain mechanisms underlying uncertainty computations. Thus, we investigated whether it is involved in disentangling these two forms of uncertainty. To this aim, we measured pupil size during a probabilistic reversal learning task. In this task, participants had to figure out which of two response options led to reward with higher probability, whereby sometimes the identity of the more advantageous response option was switched. Expected uncertainty was manipulated by varying the reward probability of the advantageous choice option, whereas the level of unexpected uncertainty was assessed by using a Bayesian computational model estimating change probability and resulting uncertainty. We found that both aspects of unexpected uncertainty influenced pupil responses, confirming that pupil-linked brain arousal is involved in model updating after unexpected changes in the environment. Furthermore, high level of expected uncertainty impeded the detection of sudden changes in the environment, both on physiological and behavioral level. These results emphasize the role of pupil-linked brain arousal and underlying neural structures in handling situations in which the previously established contingencies are no longer valid.

Characterization of striatal dopamine projections across striatal subregions in behavioral flexibility.

Behavioural flexibility is key to survival in a dynamic environmentWhile flexible, goal-directed behaviours are initially dependent on dorsomedial striatum, they become dependent on lateral striatum as behaviours become inflexible. Similarly, lesions of dopamine terminals in lateral striatum disrupt the development of inflexible habits. This work suggests that dopamine release in lateral striatum may drive inflexible behaviours, though few studies have investigated a causative role of subpopulations of striatal dopamine terminals in reversal learning, a measure of flexibility. Here, we performed two optogenetic experiments to activate dopamine terminals in dorsomedial (DMS), dorsolateral (DLS) or ventral (nucleus accumbens [NAc]) striatum in DAT-Cre mice that expressed channelrhodopsin-2 via viral injection (Experiment I) or through transgenic breeding with an Ai32 reporter line (Experiment II) to determine how specific dopamine subpopulations impact reversal learning. Mice performed a reversal task in which they self-stimulated DMS, DLS, or NAc dopamine terminals by pressing one of two levers before action-outcome lever contingencies were reversed. Largely consistent with presumed ventromedial/lateral striatal function, we found that mice self-stimulating medial dopamine terminals reversed lever preference following contingency reversal, while mice self-stimulating NAc showed parial flexibility, and DLS self-stimulation resulted in impaired reversal. Impairments in DLS mice were characterized by more regressive errors and reliance on lose-stay strategies following reversal, as well as reduced within-session learning, suggesting reward insensitivity and overreliance on previously learned actions. This study supports a model of striatal function in which DMS and ventral dopamine facilitate goal-directed responding, and DLS dopamine supports more inflexible responding.

Acute stress alters probabilistic reversal learning in healthy male adults.

Behavioural adaptation is a fundamental cognitive ability, ensuring survival by allowing for flexible adjustment to changing environments. In laboratory settings, behavioural adaptation can be measured with reversal learning paradigms requiring agents to adjust reward learning to stimulus-action-outcome contingency changes. Stress is found to alter flexibility of reward learning, but effect directionality is mixed across studies. Here, we used model-based functional MRI (fMRI) in a within-subjects design to investigate the effect of acute psychosocial stress on flexible behavioural adaptation. Healthy male volunteers (n = 28) did a reversal learning task during fMRI in two sessions, once after the Trier Social Stress Test (TSST), a validated psychosocial stress induction method, and once after a control condition. Stress effects on choice behaviour were investigated using multilevel generalized linear models and computational models describing different learning processes that potentially generated the data. Computational models were fitted using a hierarchical Bayesian approach, and model-derived reward prediction errors (RPE) were used as fMRI regressors. We found that acute psychosocial stress slightly increased correct response rates. Model comparison revealed that double-update learning with altered choice temperature under stress best explained the observed behaviour. In the brain, model-derived RPEs were correlated with BOLD signals in striatum and ventromedial prefrontal cortex (vmPFC). Striatal RPE signals for win trials were stronger during stress compared with the control condition. Our study suggests that acute psychosocial stress could enhance reversal learning and RPE brain responses in healthy male participants and provides a starting point to explore these effects further in a more diverse population.

Distinct Patterns of Abnormal Lateral Orbitofrontal Cortex Activity During Compulsive Grooming and Reversal Learning Normalize After Fluoxetine.

BACKGROUND: Patients with obsessive-compulsive disorder (OCD) display disrupted performance and abnormal lateral orbitofrontal cortex (LOFC) activity during reversal learning tasks. However, it is unknown whether compulsions and reversal learning deficits share a common neural substrate. To answer this question, we measured neural activity with in vivo calcium imaging in LOFC during compulsive grooming and reversal learning before and after fluoxetine treatment. METHODS: Sapap3 knockout (KO) mice were used as a model for OCD-relevant behaviors. Sapap3 KOs and control littermates were injected with a virus encoding GCaMP6f and implanted with gradient-index lenses to visualize LOFC activity using miniature microscopes. Grooming, reversal learning, and neural activity were measured pre- and post-fluoxetine treatment (18 mg/kg, 4 weeks). RESULTS: Baseline compulsive grooming and reversal learning impairments in KOs improved after fluoxetine treatment. In addition, KOs displayed distinct patterns of abnormal LOFC activity during grooming and reversal learning, both of which normalized after fluoxetine. Finally, reversal learning-associated neurons were distributed randomly among grooming-associated neurons (i.e., overlap is what would be expected by chance). CONCLUSIONS: In OCD, LOFC is disrupted during both compulsive behaviors and reversal learning, but whether these behaviors share common neural underpinnings is unknown. We found that LOFC plays distinct roles in compulsive grooming and impaired reversal learning and their improvement with fluoxetine. These findings suggest that LOFC plays separate roles in pathophysiology and treatment of different perseverative behaviors in OCD.

Dorsal posterior cingulate cortex responds to negative feedback information supporting learning and relearning of response policies.

Many challenges in life come without explicit instructions. Instead, humans need to test, select, and adapt their behavioral responses based on feedback from the environment. While reward-centric accounts of feedback processing primarily stress the reinforcing aspect of positive feedback, feedback's central function from an information-processing perspective is to offer an opportunity to correct errors, thus putting a greater emphasis on the informational content of negative feedback. Independent of its potential rewarding value, the informational value of performance feedback has recently been suggested to be neurophysiologically encoded in the dorsal portion of the posterior cingulate cortex (dPCC). To further test this association, we investigated multidimensional categorization and reversal learning by comparing negative and positive feedback in an event-related functional magnetic resonance imaging experiment. Negative feedback, compared with positive feedback, increased activation in the dPCC as well as in brain regions typically involved in error processing. Only in the dPCC, subarea d23, this effect was significantly enhanced in relearning, where negative feedback signaled the need to shift away from a previously established response policy. Together with previous findings, this result contributes to a more fine-grained functional parcellation of PCC subregions and supports the dPCC's involvement in the adaptation to behaviorally relevant information from the environment.

Modulation of ventromedial orbitofrontal cortical glutamatergic activity affects the explore-exploit balance and influences value-based decision-making.

The balance between exploration and exploitation is essential for decision-making. The present study investigated the role of ventromedial orbitofrontal cortex (vmOFC) glutamate neurons in mediating value-based decision-making by first using optogenetics to manipulate vmOFC glutamate activity in rats during a probabilistic reversal learning (PRL) task. Rats that received vmOFC activation during informative feedback completed fewer reversals and exhibited reduced reward sensitivity relative to rats. Analysis with a Q-learning computational model revealed that increased vmOFC activity did not affect the learning rate but instead promoted maladaptive exploration. By contrast, vmOFC inhibition increased the number of completed reversals and increased exploitative behavior. In a separate group of animals, calcium activity of vmOFC glutamate neurons was recorded using fiber photometry. Complementing our results above, we found that suppression of vmOFC activity during the latter part of rewarded trials was associated with improved PRL performance, greater win-stay responding and selecting the correct choice on the next trial. These data demonstrate that excessive vmOFC activity during reward feedback disrupted value-based decision-making by increasing the maladaptive exploration of lower-valued options. Our findings support the premise that pharmacological interventions that normalize aberrant vmOFC glutamate activity during reward feedback processing may attenuate deficits in value-based decision-making.

Striatal dopamine dissociates methylphenidate effects on value-based versus surprise-based reversal learning.

Psychostimulants such as methylphenidate are widely used for their cognitive enhancing effects, but there is large variability in the direction and extent of these effects. We tested the hypothesis that methylphenidate enhances or impairs reward/punishment-based reversal learning depending on baseline striatal dopamine levels and corticostriatal gating of reward/punishment-related representations in stimulus-specific sensory cortex. Young healthy adults (N = 100) were scanned with functional magnetic resonance imaging during a reward/punishment reversal learning task, after intake of methylphenidate or the selective D2/3-receptor antagonist sulpiride. Striatal dopamine synthesis capacity was indexed with [18F]DOPA positron emission tomography. Methylphenidate improved and sulpiride decreased overall accuracy and response speed. Both drugs boosted reward versus punishment learning signals to a greater degree in participants with higher dopamine synthesis capacity. By contrast, striatal and stimulus-specific sensory surprise signals were boosted in participants with lower dopamine synthesis. These results unravel the mechanisms by which methylphenidate gates both attention and reward learning.

Phasic Dopamine Changes and Hebbian Mechanisms during Probabilistic Reversal Learning in Striatal Circuits: A Computational Study.

Cognitive flexibility is essential to modify our behavior in a non-stationary environment and is often explored by reversal learning tasks. The basal ganglia (BG) dopaminergic system, under a top-down control of the pre-frontal cortex, is known to be involved in flexible action selection through reinforcement learning. However, how adaptive dopamine changes regulate this process and learning mechanisms for training the striatal synapses remain open questions. The current study uses a neurocomputational model of the BG, based on dopamine-dependent direct (Go) and indirect (NoGo) pathways, to investigate reinforcement learning in a probabilistic environment through a task that associates different stimuli to different actions. Here, we investigated: the efficacy of several versions of the Hebb rule, based on covariance between pre- and post-synaptic neurons, as well as the required control in phasic dopamine changes crucial to achieving a proper reversal learning. Furthermore, an original mechanism for modulating the phasic dopamine changes is proposed, assuming that the expected reward probability is coded by the activity of the winner Go neuron before a reward/punishment takes place. Simulations show that this original formulation for an automatic phasic dopamine control allows the achievement of a good flexible reversal even in difficult conditions. The current outcomes may contribute to understanding the mechanisms for active control of dopamine changes during flexible behavior. In perspective, it may be applied in neuropsychiatric or neurological disorders, such as Parkinson's or schizophrenia, in which reinforcement learning is impaired.